feat(analytics): track feature enablement & usage (task-209)#1008
Merged
feat(analytics): track feature enablement & usage (task-209)#1008
Conversation
Extends the consent-gated GA4 pipeline with feature-enablement snapshots and per-prompt feature_used events. Privacy guarantees (task-206/208) are preserved: closed per-event param allowlists, enum-typed feature_ids, coarse bucketed counts, shape/length rejection, fire-and-forget. Foundation - AnalyticsEventBuilder: generic (eventName, Map) -> JSON builder with closed per-event allowlist, enum-value allowlist, path/URL/newline shape rejection, and 128-char length cap. - FeatureId enum (closed set) with usage-only flag for rejecting non-snapshot-eligible features at trackFeatureEnabled. - ProviderType enum: LOCAL/CLOUD/NONE with OPTIONAL folded into CLOUD. - Buckets utility: standard and chat-memory ladders with boundary tests. - AnalyticsService refactored to route all events through the builder; existing trackPromptExecuted/trackModelSelected behavior preserved (AnalyticsServiceTest still green). Session snapshot - DevoxxGenieSettingsChangedTopic: MessageBus topic for settings changes. - AnalyticsSessionSnapshotService: APP-level @service, AtomicBoolean- guarded one-shot per IDE session. Subscribes to the topic to re-arm, so any settings change triggers a fresh snapshot. Narrow FeatureEventSink interface keeps AnalyticsService final. - Wired from PostStartupActivity (idempotent across project opens), AnalyticsConsentNotifier "Keep Enabled" action, and GeneralSettingsComponent#apply (via the topic). Feature usage instrumentation - ChatMessageContext gains projectContextFullUsed, projectContextSelectedUsed, devoxxGenieMdUsed booleans and a final AtomicInteger mcpCallCount. - InstrumentedMcpToolProvider: counts real ToolExecutor.execute() invocations inside the wrapped executor (not provideTools), so speculative tool-list calls and denied/filtered tools never inflate counts. - MCPExecutionService: new optional-counter overloads preserve the Approval -> Instrumented -> Filtered -> raw stack order. - FeatureUsageTracker: static facade emitting feature_used per activated feature; enforces "tool_call_count only meaningful for agent/mcp". - PromptExecutionService: calls FeatureUsageTracker.emitForPrompt in the task.whenComplete hook after the strategy finishes. - StreamingPromptStrategy and NonStreamingPromptExecutionService: pass context.getMcpCallCount() into the MCP provider chain and emit agent feature_used with the AgentLoopTracker's final count on success, error, or cancellation. Tests (30 new + 12 existing analytics cases stay green) - BucketsTest, ProviderTypeTest, AnalyticsEventBuilderTest, AnalyticsSessionSnapshotServiceTest, InstrumentedMcpToolProviderTest. Also carries two small task-208 follow-ups that were on the branch: Taskfile.yml gains a `version` task, and the General settings panel is renamed to "Analytics" to match the panel's scope. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…sk-209, phase 4) Completes task-209 by setting the ChatMessageContext analytics booleans at their assembly sites, instrumenting SemanticSearchService and SubAgentRunner, updating the three disclosure surfaces in lockstep, and shipping the shared GA4 schema doc. Context set-sites (AC #25) - ChatMessageContextUtil.setWindowContext(): flips projectContextFullUsed when full project is attached; processAttachedFiles() flips projectContextSelectedUsed when non-empty attached files are queued. - ChatMemoryManager.buildSystemPrompt(context): mirrors the useDevoxxGenieMdInPrompt + file-present gate to set devoxxGenieMdUsed. Semantic search (AC #20) - SemanticSearchService.search() now fires FeatureUsageTracker .semanticSearchUsed at invocation — query text never leaves the IDE. Sub-agent analytics (AC #23 subtlety) - SubAgentRunner emits its own agent feature_used event in a finally block so success/error/cancellation all go through. Uses ProviderType.fromModelProvider derived from resolvedProviderName, with a safe fallback to ProviderType.NONE on unknown provider strings. Sub- agent events are separate from parent events so the parent isn't double-counted. Disclosure lockstep (ACs #11, #27) - AnalyticsConsentNotifier: new first-run bullets covering feature enablement snapshot + per-prompt usage, and the "never sent" list is extended with MCP server names/URLs/commands/tool names and custom prompt names/bodies. - GeneralSettingsComponent: matching sent/not-sent bullets in the settings panel. - plugin.xml: marketplace description updated to match, and the Settings path is now "DevoxxGenie → Analytics" (matching the existing panel rename). Schema doc (AC #13) - docs/analytics-schema.md: single source of truth for the GA4 schema shared with GenieBuilder — event shapes, common params, closed feature_id enum, provider_type mapping (including OPTIONAL → cloud), both bucket ladders, and a change-process checklist. GenieBuilder follow-up (AC #14) - Filed task-210 in this repo's backlog as the admin-UI follow-up, with dependency → task-209, closed enum references, and explicit out-of-scope rules (never display free-form MCP server or custom prompt values). Tests - FeatureUsageTrackerTest: guardrails asserting every FeatureId wire value matches the schema doc, the usage-only flag is set on exactly the four documented ids, fromWireValue round-trips, and semanticSearchUsed(null) is fail-silent. Task-209 ACs are 27/28. AC #12's explicit task-208 offline-fire-and- forget regression is covered implicitly by the existing AnalyticsServiceTest.asyncNetworkFailureIsSilent which now runs through the refactored AnalyticsEventBuilder path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All findings from the second review round are fixed and the full test suite is green. #1 MCP-inside-agent not counted - AgentToolProviderFactory.createToolProvider now has an overload taking an optional AtomicInteger mcpCallCounter and threads it into MCPExecutionService.createRawMCPToolProvider(counter), so agent-hosted MCP invocations also go through InstrumentedMcpToolProvider. - StreamingPromptStrategy and NonStreamingPromptExecutionService pass context.getMcpCallCount() into the new overload. #2 RAG/WebSearch activation flags were never set on ChatMessageContext - SearchOptionsPanel writes ragActivated/webSearchActivated to DevoxxGenieStateService but ChatMessageContext.ragActivated/ webSearchActivated were always false. ChatMessageContextUtil.createContext now mirrors both flags from state onto the context so FeatureUsageTracker.emitForPrompt fires the rag and web_search_* events as designed. #3 Snapshot one-shot consumed before consent - AnalyticsSessionSnapshotService.snapshotIfNeeded now preflights analyticsNoticeAcknowledged + analyticsEnabled BEFORE compareAndSet(false, true). The first opted-in session still emits after "OK, Keep Enabled" is clicked because the guard never fired. #4 Settings-change re-arm missing from RAG/MCP/WebSearch/Agent panels - Extracted a fail-silent helper DevoxxGenieSettingsChangedTopic.notifySettingsChanged() and wired it into every relevant apply() method: GeneralSettingsComponent, RAGSettingsConfigurable, MCPSettingsComponent, WebSearchProviders- Configurable, AgentSettingsComponent. The helper swallows any exception (including null MessageBus in test environments) so settings apply() paths never crash because analytics is unreachable. #5 Windows drive-letter absolute paths not rejected - AnalyticsEventBuilder.rejectShape now also drops values matching the pattern `[A-Za-z]:[/\\].*`, plus four new unit tests covering backslash, forward-slash, lowercase drive letters, and the existing UNC path case. #6 Semantic search usage emitted provider_type=none - Moved the FeatureUsageTracker.semanticSearchUsed call out of SemanticSearchService.search and into MessageCreationService where the active LanguageModel is in scope. The event now reflects the real provider. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
475ec18 to
2ec0a63
Compare
All 28 acceptance criteria satisfied; final summary captured. See the three feat/fix commits on this branch for the implementation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2ec0a63 to
7591a00
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
feature_enabled(once per IDE session per enabled feature),feature_used(one per activated feature per prompt), andfeature_counts(one per session with bucketed MCP server / custom prompt / chat memory counts).AnalyticsEventBuilderwith closed per-event allowlists, enum-typed params, shape/length rejection (absolute paths incl. Windows drive letters, URLs, newlines, >128 chars).AnalyticsSessionSnapshotService(APP-level@Service) guards emission with anAtomicBooleanand preflights consent before burning the one-shot.InstrumentedMcpToolProvidercounts realToolExecutor.execute()calls in stack orderApproval → Instrumented → Filtered → raw, covering both standalone MCP and MCP-inside-agent via a threaded counter. Agent events fire fromStreamingPromptStrategy,NonStreamingPromptExecutionService, andSubAgentRunner.AnalyticsConsentNotifier,GeneralSettingsComponent, andplugin.xml. Shared schema documented atdocs/analytics-schema.mdas the source of truth for both this repo and../GenieBuilder. Follow-up tasks filed:task-210here andtask-197in GenieBuilder (with the exactTRACKED_EVENTS/EVENT_LABELS/EVENT_CATEGORIESedits and the six GA4 custom dimensions to register).Test plan
./gradlew test— full suite green (~35 analytics-focused tests acrossAnalyticsServiceTest,AnalyticsEventBuilderTest,AnalyticsSessionSnapshotServiceTest,BucketsTest,ProviderTypeTest,InstrumentedMcpToolProviderTest,FeatureUsageTrackerTest)AnalyticsServiceTest(12 cases) remains green after thebuildPayloadrefactor — behavior-preserving forprompt_executed/model_selected/Users/...), URLs (https://...), Windows drive letters (C:\...,D:/..., lowercase), newlines, overly long values — all unit-testedsnapshotIfNeeded()called twice → single emission; re-arm viasettingsChanged()→ second emission — unit-tested without IntelliJ platform fixturessemantic_search,project_context_full,project_context_selected,devoxxgenie_md) rejected if passed totrackFeatureEnabled— unit-tested./gradlew runIde: toggle RAG / MCP / Agent / Web Search in Settings and verify the GA4 endpoint receives a freshfeature_enabledsnapshot on apply (requires registering the 6 new GA4 custom dimensions first — seetask-197in GenieBuilder)app_name=devoxxgenie-intellij🤖 Generated with Claude Code